Purpose

This document lays out results for estimating statistical power and minimum detectable effect size (MDES) for: Blocked RCT, with 2 levels, and randomization done at level 1 (individual level). This document contains models assuming constant treatment effects, fixed treatment effects, and random treatment effects.

We compare results of the derived methods against Monte Carlo Simulations & the PowerUp R package on power. The methods are derived by Kristin Porter as outlined in the paper/s here (Add reference section).

Power Validation

In this section, we validate power results for different definitions of power and different adjustment procedures.

If you are previewing this in html, please click on the code section to review more detailed parameters.

This section sets up the simulation-level parameters, such as how many monte carlo samples to draw.

Setup 1: School size 100

Main parameters:

n.j = 100
M = 3
J = 20
rho = (0.5, 0.5, 0.5)
MDES = (0.125, 0.125, 0.125)
R2.1 = (0, 0, 0)
R2.2 = (0, 0, 0)
ICC.2 = (0, 0, 0)
Omega.2 = 0.5

Individual Statistical power is estimated to be around 0.8. Please check the table below for estimations of other definitions of statistical power. Across all power definitions, estimation results across the PUMP package, Monte Carlo Simulation results and PowerUp package are about the same as detailed below.

Setup 2: School size 75

n.j = 75

Setup 3: School size 50

n.j = 50

Varying R2

R2.1 = 0.6, 0.6, 0.6

R2.2 = 0.6, 0.6, 0.6

Varying rho

rho.default = 0.2

rho.default = 0.8

Varying true positives

MDES = 0.125, 0, 0

Varying ICC for relevant models

ICC.2 = c(0.8, 0.8, 0.8)

MDES validation

In this section, we validate the derived minimum detectable effect size (MDES) function across different MTP (Bonferroni, Benjamini-Hocheberg and Holms) and power definition (individual and minimum_1). The validation is carried out by generating power values for the specific MTP and power definitions for a specific MDES. Then, we supply the generated estimate power as an input parameter to see if the specified MDES is returned.

Power: 0.65 - 0.87

Below, we have input parameters of number of outcomes, minimum detectable effect size, statistical power, and significance level that would estimate a high statistical power value of above 0.65 to 1.00 for different definitions of power. This estimated power value is used as an input to the mdes function where we validate if the returned mdes value is close to the mdes value that was inputted to provide the estimated power.

Individual Power

School size: 50

The Bonferroni, Benjamini Hocheberg and Holms’ adjusted MDES and the targeted MDES are within an acceptable margin of error for the specified targetted power. The adjusted MDES is estimated at the corresponding individual power in the table below. The corresponding individual power is the power at which our MDES method stops looking for a MDES value close to the target MDES.

Sample size validation